Add log discovery based on service information in WLM by vitkyrka · Pull Request #38006 · DataDog/datadog-agent

vitkyrka · 2025-06-17T13:42:01Z

What does this PR do?

Add an autodiscovery provider which listens for new processes/services from WorkloadMeta and schedules log configs when they have logs to tail.

Note that the provider needs to be enabled via a config change, it is not enabled by default.

Motivation

https://datadoghq.atlassian.net/wiki/spaces/DSCVR/pages/5212537551
https://datadoghq.atlassian.net/browse/DSCVR-165

Describe how you validated your changes

Test included in the PR.

Also tested manually (see linked wiki page), that currently requires code changes to enable the process collector.

Possible Drawbacks / Trade-offs

Additional Notes

cit-pr-commenter · 2025-06-17T13:59:21Z

Go Package Import Differences

Baseline: a3c28ec
Comparison: c6cef08

binary	os	arch	change
cluster-agent-cloudfoundry	linux	amd64	+2, -0 +github.com/hashicorp/golang-lru/v2/internal +github.com/hashicorp/golang-lru/v2/simplelru
cluster-agent-cloudfoundry	linux	arm64	+2, -0 +github.com/hashicorp/golang-lru/v2/internal +github.com/hashicorp/golang-lru/v2/simplelru

agent-platform-auto-pr · 2025-06-17T15:40:54Z

Static quality checks

✅ Please find below the results from static quality gates
Comparison made with ancestor a3c28ec

Successful checks

Info

	Quality gate	Delta	On disk size (MiB)	Delta	On wire size (MiB)
✅	agent_deb_amd64	$${+0.02}$$	$${699.66}$$ < $${704.84}$$	$${-0}$$	$${177.21}$$ < $${178.58}$$
✅	agent_deb_amd64_fips	$${+0.02}$$	$${697.91}$$ < $${703.09}$$	$${-0.02}$$	$${176.55}$$ < $${178.12}$$
✅	agent_heroku_amd64	$${+0.03}$$	$${349.74}$$ < $${355.37}$$	$${+0.02}$$	$${93.82}$$ < $${95.72}$$
✅	agent_msi	$${+0.16}$$	$${974.59}$$ < $${979.61}$$	$${+0.01}$$	$${147.62}$$ < $${149.31}$$
✅	agent_rpm_amd64	$${+0.02}$$	$${699.65}$$ < $${704.83}$$	$${+0.04}$$	$${178.77}$$ < $${180.22}$$
✅	agent_rpm_amd64_fips	$${+0.02}$$	$${697.9}$$ < $${703.08}$$	$${+0.01}$$	$${178.47}$$ < $${179.85}$$
✅	agent_rpm_arm64	$${+0.02}$$	$${688.63}$$ < $${694.74}$$	$${-0.03}$$	$${161.78}$$ < $${163.96}$$
✅	agent_rpm_arm64_fips	$${+0.02}$$	$${686.93}$$ < $${693.05}$$	$${-0}$$	$${160.88}$$ < $${163.0}$$
✅	agent_suse_amd64	$${+0.02}$$	$${699.65}$$ < $${704.83}$$	$${+0.04}$$	$${178.77}$$ < $${180.22}$$
✅	agent_suse_amd64_fips	$${+0.02}$$	$${697.9}$$ < $${703.08}$$	$${+0.01}$$	$${178.47}$$ < $${179.85}$$
✅	agent_suse_arm64	$${+0.02}$$	$${688.63}$$ < $${694.74}$$	$${-0.03}$$	$${161.78}$$ < $${163.96}$$
✅	agent_suse_arm64_fips	$${+0.02}$$	$${686.93}$$ < $${693.05}$$	$${-0}$$	$${160.88}$$ < $${163.0}$$
✅	docker_agent_amd64	$${+0.02}$$	$${781.38}$$ < $${788.65}$$	$${+0.02}$$	$${269.3}$$ < $${272.01}$$
✅	docker_agent_arm64	$${+0.02}$$	$${793.69}$$ < $${802.0}$$	$${-0}$$	$${256.1}$$ < $${259.7}$$
✅	docker_agent_jmx_amd64	$${+0.02}$$	$${972.68}$$ < $${979.84}$$	$${+0.01}$$	$${338.33}$$ < $${340.95}$$
✅	docker_agent_jmx_arm64	$${+0.02}$$	$${973.59}$$ < $${981.8}$$	$${+0.02}$$	$${321.12}$$ < $${324.65}$$
✅	docker_cluster_agent_amd64	$${+0.04}$$	$${214.05}$$ < $${214.5}$$	$${+0.02}$$	$${72.76}$$ < $${73.51}$$
✅	docker_cluster_agent_arm64	$${0}$$	$${229.88}$$ < $${230.33}$$	$${+0.02}$$	$${69.01}$$ < $${69.77}$$
✅	docker_cws_instrumentation_amd64	$${0}$$	$${7.07}$$ < $${7.12}$$	$${-0}$$	$${2.95}$$ < $${3.29}$$
✅	docker_cws_instrumentation_arm64	$${0}$$	$${6.69}$$ < $${6.92}$$	$${+0}$$	$${2.7}$$ < $${3.07}$$
✅	docker_dogstatsd_amd64	$${0}$$	$${38.54}$$ < $${39.57}$$	$${+0}$$	$${14.89}$$ < $${15.76}$$
✅	docker_dogstatsd_arm64	$${0}$$	$${37.23}$$ < $${38.2}$$	$${-0}$$	$${14.34}$$ < $${14.83}$$
✅	dogstatsd_deb_amd64	$${0}$$	$${29.77}$$ < $${31.4}$$	$${+0}$$	$${7.85}$$ < $${8.95}$$
✅	dogstatsd_deb_arm64	$${0}$$	$${28.38}$$ < $${29.97}$$	$${-0}$$	$${6.81}$$ < $${7.89}$$
✅	dogstatsd_rpm_amd64	$${0}$$	$${29.77}$$ < $${31.4}$$	$${+0}$$	$${7.86}$$ < $${8.96}$$
✅	dogstatsd_suse_amd64	$${0}$$	$${29.77}$$ < $${31.4}$$	$${+0}$$	$${7.86}$$ < $${8.96}$$
✅	iot_agent_deb_amd64	$${+0.03}$$	$${54.15}$$ < $${54.55}$$	$${+0}$$	$${13.65}$$ < $${14.45}$$
✅	iot_agent_deb_arm64	$${+0.03}$$	$${51.47}$$ < $${51.9}$$	$${+0.01}$$	$${11.81}$$ < $${12.63}$$
✅	iot_agent_deb_armhf	$${+0.03}$$	$${50.98}$$ < $${51.42}$$	$${+0.03}$$	$${11.92}$$ < $${12.74}$$
✅	iot_agent_rpm_amd64	$${+0.03}$$	$${54.15}$$ < $${54.55}$$	$${+0.01}$$	$${13.67}$$ < $${14.47}$$
✅	iot_agent_rpm_arm64	$${+0.03}$$	$${51.47}$$ < $${51.91}$$	$${+0.01}$$	$${11.83}$$ < $${12.65}$$
✅	iot_agent_suse_amd64	$${+0.03}$$	$${54.15}$$ < $${54.55}$$	$${+0.01}$$	$${13.67}$$ < $${14.47}$$

cit-pr-commenter · 2025-07-03T14:07:31Z

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: d0f37809-8fff-47fe-be40-3fdd7d0deac7

Baseline: a3c28ec
Comparison: c6cef08
Diff

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	docker_containers_cpu	% cpu utilization	+3.69	[+0.60, +6.78]	1	Logs

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	docker_containers_cpu	% cpu utilization	+3.69	[+0.60, +6.78]	1	Logs
➖	ddot_metrics	memory utilization	+0.39	[+0.27, +0.51]	1	Logs
➖	ddot_logs	memory utilization	+0.33	[+0.24, +0.42]	1	Logs
➖	quality_gate_idle	memory utilization	+0.17	[+0.13, +0.20]	1	Logs bounds checks dashboard
➖	file_tree	memory utilization	+0.07	[+0.03, +0.11]	1	Logs
➖	uds_dogstatsd_20mb_12k_contexts_20_senders	memory utilization	+0.06	[-0.00, +0.12]	1	Logs
➖	file_to_blackhole_100ms_latency	egress throughput	+0.02	[-0.59, +0.64]	1	Logs
➖	uds_dogstatsd_to_api	ingress throughput	+0.01	[-0.31, +0.32]	1	Logs
➖	file_to_blackhole_0ms_latency	egress throughput	-0.00	[-0.57, +0.57]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	-0.00	[-0.02, +0.02]	1	Logs
➖	quality_gate_idle_all_features	memory utilization	-0.01	[-0.06, +0.05]	1	Logs bounds checks dashboard
➖	file_to_blackhole_500ms_latency	egress throughput	-0.01	[-0.67, +0.64]	1	Logs
➖	file_to_blackhole_1000ms_latency	egress throughput	-0.04	[-0.61, +0.53]	1	Logs
➖	otlp_ingest_metrics	memory utilization	-0.09	[-0.23, +0.04]	1	Logs
➖	otlp_ingest_logs	memory utilization	-0.24	[-0.36, -0.11]	1	Logs
➖	docker_containers_memory	memory utilization	-0.40	[-0.46, -0.33]	1	Logs
➖	quality_gate_logs	% cpu utilization	-0.76	[-3.53, +2.01]	1	Logs bounds checks dashboard
➖	quality_gate_metrics_logs	memory utilization	-1.53	[-1.92, -1.14]	1	Logs bounds checks dashboard
➖	tcp_syslog_to_blackhole	ingress throughput	-2.65	[-2.71, -2.59]	1	Logs

Bounds Checks: ✅ Passed

perf	experiment	bounds_check_name	replicates_passed	links
✅	docker_containers_cpu	simple_check_run	10/10
✅	docker_containers_memory	memory_usage	10/10
✅	docker_containers_memory	simple_check_run	10/10
✅	file_to_blackhole_0ms_latency	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency	memory_usage	10/10
✅	file_to_blackhole_100ms_latency	lost_bytes	10/10
✅	file_to_blackhole_100ms_latency	memory_usage	10/10
✅	file_to_blackhole_500ms_latency	lost_bytes	10/10
✅	file_to_blackhole_500ms_latency	memory_usage	10/10
✅	quality_gate_idle	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_idle	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_idle_all_features	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_idle_all_features	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_logs	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_logs	lost_bytes	10/10	bounds checks dashboard
✅	quality_gate_logs	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_metrics_logs	cpu_usage	10/10	bounds checks dashboard
✅	quality_gate_metrics_logs	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_metrics_logs	lost_bytes	10/10	bounds checks dashboard
✅	quality_gate_metrics_logs	memory_usage	10/10	bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.

quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.

Add an autodiscovery provider which listens for new services from WorkloadMeta and schedules log configs when they have logs to tail.

vitkyrka · 2025-07-15T12:52:47Z

The extra imports in otel-agent will be handled by #38784

adel121 · 2025-07-30T10:55:21Z

+			// Ignore containers since log files inside them usually can't be
+			// accessed from here since they are in a different namespace.


Question:

If I understand correctly, the logs agent accesses logs by looking at files under /var/log/pods

We also have /var/log/containers/

So why is being in a different namespace an issue when it comes to collecting logs? (My thought is that /var/log/containers/ are host level files that are not scoped to namespaces, so in theory they should be accessible by the log agent)

You're correct that container runtimes store the stderr/stdout from containers in paths like /var/log/containers on the host, but in this case we are ignoring logs files written inside containers (not via the container's stdout/stderr). See also https://datadoghq.atlassian.net/wiki/spaces/DSCVR/pages/5212537551/#Log-files-inside-containers

makes sense

Thanks for the clarification.

adel121 · 2025-07-30T11:11:42Z

+func (p *processLogConfigProvider) generateServiceLogKey(logPath string) string {
+	return strings.ReplaceAll(logPath, "/", "_")
+}


Just to be sure:

I assume this should return a unique key (correct me if I wrong).

Isn't there a chance for collision here?

like if we had something like:

some-dir/logs/one

and

some-dire_log_one

Both will get the same key.

You're right. I realized there's no need to do this massaging of the path so I just used the raw path now which should fix this.

adel121

Looks good 👍

Left 2 minor comments/questions.

vitkyrka · 2025-07-31T12:16:06Z

/merge

dd-devflow-routing-codex · 2025-07-31T12:16:10Z

View all feedbacks in Devflow UI.

2025-07-31 12:16:09 UTC ℹ️ Start processing command /merge

2025-07-31 12:16:20 UTC ℹ️ MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2025-07-31 12:29:17 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in main is approximately 52m (p90).

2025-07-31 12:58:28 UTC ❌ MergeQueue: The build pipeline contains failing jobs for this merge request

Build pipeline has failing jobs for d152123:

⚠️ Do NOT retry failed jobs directly (why?).

What to do next?

Investigate the failures and when ready, re-add your pull request to the queue!
If your PR checks are green, try to rebase/merge. It might be because the CI run is a bit old.
Any question, go check the FAQ.

Details

Since those jobs are not marked as being allowed to fail, the pipeline will most likely fail.
Therefore, and to allow other builds to be processed, this merge request has been rejected and the pipeline got canceled.

vitkyrka · 2025-08-01T10:37:46Z

/merge

dd-devflow-routing-codex · 2025-08-01T10:37:49Z

View all feedbacks in Devflow UI.

2025-08-01 10:37:49 UTC ℹ️ Start processing command /merge

2025-08-01 10:37:54 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 54m (p90).

2025-08-01 11:14:45 UTC ℹ️ MergeQueue: This merge request was merged

vitkyrka added changelog/no-changelog No changelog entry needed qa/done QA done before merge and regressions are covered by tests team/agent-discovery labels Jun 17, 2025

vitkyrka force-pushed the vitkyrka/disco-logs branch from b661cec to 5975816 Compare July 3, 2025 13:21

github-actions Bot added component/system-probe long review PR is complex, plan time to review it labels Jul 3, 2025

vitkyrka force-pushed the vitkyrka/disco-logs branch from 5975816 to 0659734 Compare July 4, 2025 11:01

vitkyrka force-pushed the vitkyrka/disco-logs branch from daba728 to ccf3db6 Compare July 11, 2025 17:02

github-actions Bot removed the component/system-probe label Jul 11, 2025

Add logs discovery based on SD data

b81f182

Add an autodiscovery provider which listens for new services from WorkloadMeta and schedules log configs when they have logs to tail.

vitkyrka force-pushed the vitkyrka/disco-logs branch from ccf3db6 to b81f182 Compare July 15, 2025 08:29

vitkyrka mentioned this pull request Jul 15, 2025

Move GetWorkloadmetaInit to separate package #38784

Merged

vitkyrka changed the title ~~Add log discovery based on SD~~ Add log discovery based on service information in WLM Jul 15, 2025

vitkyrka marked this pull request as ready for review July 15, 2025 13:07

vitkyrka requested review from a team as code owners July 15, 2025 13:07

vitkyrka added the ask-review Ask in slack required teams to review this PR label Jul 15, 2025

davidor reviewed Jul 15, 2025

View reviewed changes

vitkyrka added 7 commits July 17, 2025 16:06

Ack before process

beaf3d0

Add code owners for process_log*

c40e356

Remove log in test

b720079

build tests on windows and skip specific

0946587

map

8ed6644

Use [Not]Contains

1bf8430

Merge remote-tracking branch 'origin/main' into vitkyrka/disco-logs

e519b37

vitkyrka requested a review from a team July 21, 2025 08:54

ddrthall approved these changes Jul 23, 2025

View reviewed changes

vitkyrka requested a review from davidor July 24, 2025 07:23

adel121 reviewed Jul 30, 2025

View reviewed changes

vitkyrka added 5 commits July 31, 2025 10:17

Use paths for service ids

5c7af18

Use paths for integration name

2b3081b

Dedup log paths in test

5292c36

Merge remote-tracking branch 'origin/main' into vitkyrka/disco-logs

abee7f6

Linter

682e88f

vitkyrka requested a review from adel121 July 31, 2025 10:46

adel121 approved these changes Jul 31, 2025

View reviewed changes

vitkyrka added 2 commits July 31, 2025 17:18

Merge remote-tracking branch 'origin/main' into vitkyrka/disco-logs

f41197b

Ignore pkgconfigusage linter for new file

c6cef08

vitkyrka requested a review from a team as a code owner July 31, 2025 15:27

CelianR approved these changes Aug 1, 2025

View reviewed changes

dd-mergequeue Bot merged commit 4290a07 into main Aug 1, 2025
263 checks passed

dd-mergequeue Bot deleted the vitkyrka/disco-logs branch August 1, 2025 11:14

github-actions Bot added this to the 7.70.0 milestone Aug 1, 2025

frank-spano pushed a commit that referenced this pull request Aug 11, 2025

Add log discovery based on service information in WLM (#38006)

2b52f4b

		// Ignore containers since log files inside them usually can't be
		// accessed from here since they are in a different namespace.

Uh oh!

Conversation

vitkyrka commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Describe how you validated your changes

Possible Drawbacks / Trade-offs

Additional Notes

Uh oh!

cit-pr-commenter Bot commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Go Package Import Differences

Uh oh!

agent-platform-auto-pr Bot commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Static quality checks

Info

Uh oh!

cit-pr-commenter Bot commented Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Regression Detector

Regression Detector Results

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Fine details of change detection per experiment

Bounds Checks: ✅ Passed

Explanation

CI Pass/Fail Decision

Uh oh!

vitkyrka commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adel121 Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

vitkyrka Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

adel121 Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

adel121 Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

vitkyrka Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

adel121 left a comment

Choose a reason for hiding this comment

Uh oh!

vitkyrka commented Jul 31, 2025

Uh oh!

dd-devflow-routing-codex Bot commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What to do next?

Uh oh!

vitkyrka commented Aug 1, 2025

Uh oh!

dd-devflow-routing-codex Bot commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vitkyrka commented Jun 17, 2025 •

edited

Loading

cit-pr-commenter Bot commented Jun 17, 2025 •

edited

Loading

agent-platform-auto-pr Bot commented Jun 17, 2025 •

edited

Loading

cit-pr-commenter Bot commented Jul 3, 2025 •

edited

Loading

vitkyrka commented Jul 15, 2025 •

edited

Loading

dd-devflow-routing-codex Bot commented Jul 31, 2025 •

edited

Loading

dd-devflow-routing-codex Bot commented Aug 1, 2025 •

edited

Loading